The trend towards distributed computing, coupled with the pervasiveness of the Internet, has led to a decentralization of resources, such as files and programs, for users and system administrators. As this trend of decentralization continues, user information and data has the potential of being stored on servers and computers remotely located all over the world. As this decentralization expands, system administrators have the task of monitoring and updating computers spread over great distances. The task of monitoring and maintaining these computers is physically trying, if not impossible without a method of easily accessing and controlling the remotely located computers.
To this end, hardware and software solutions have been developed which allow users to access and control computers remotely. Early solutions included software programs that allowed text based control of remotely located computers. An example of this would be a user running a telnet program on a simple Windows-based computer to access files and run programs on a UNIX server. In this implementation, a telnet server or “daemon” is installed and started on the UNIX server. The daemon continually runs on the machine searching for and responding to new requests. A user wishing to access information on that machine starts a telnet client program which allows the user to issue a request to the daemon. After verification of the user's identity, the user has access to all of or a portion of the information on the accessed remote computer. The method is useful, but in many instances has limitations and many drawbacks.
For example, in a Windows-based computer with a telnet operation, the telnet access is dependent upon the server not crashing and continually running the telnet daemon. If the server fails, crashes, or stops this daemon, a system administrator must physically restart the remote computer or the daemon on-site. Thus, this scheme is reliant on both a robust server and a robust daemon. Furthermore, the telnet programs are normally limited to text.
More advanced software programs have been developed that allow for graphical user interfaces and greater degrees of control. Examples include Windows® XP® remote desktop, and common PCAnywhere® programs. In many of these solutions, the user can control and view the remote computer, as if it were local, with full control of the mouse and keyboard. However, like the telnet scheme, these solutions rely on software running on both the client computer and the server computer device. Specifically, the server has a daemon program similar to the daemon used in the telnet scheme. If the daemon fails, the local computer will lose control of the remote computer. Like the telnet solution, these graphical solutions still rely on software and are thus faced with substantial limitations.
Another major drawback of these software solutions is the consumption of processing power on the remote computer. Specifically, the daemon program requires resources such as memory and microprocessor execution time from the server. In addition, once the connection is established, these solutions normally use the remote computer's existing modem or Internet connection. Thus, these software solutions consume a substantial portion of the bandwidth available to the server. Both the bandwidth consumption and the power consumption can severely degrade the performance of the server.
In addition, the server software does not allow the system administrator full access to the remote computer at all times. For example, while the computer is rebooting and starting the operating system, the daemon program is not running. Therefore, the system administrator does not have access to the server during these periods. This is a major pitfall especially if the system administrator wishes to view or edit BIOS settings or view the server restart.
To avoid the aforementioned pitfalls of these software solutions, system administrators use hardware solutions which are less reliant on the remote server in order to function. For example, keyboard, video, and mouse (“KVM”) switches have been developed that allow a single keyboard, video, and mouse to control multiple computers. The computers are often remotely located from the user or system administrator's computer (i.e., the local computer). These switches route the keyboard and mouse signals of the user computer to one of the remotely located computers chosen by the user. Similarly, the video output of the chosen computer is routed to the attached local monitor. Generally the user is able to switch to any of a series of remote computers.
A KVM switch is useful for many reasons. For example, if a user has many computers, and wants to save space or cost by eliminating extra mice, keyboards, and monitors for each remote computers. The cost and space saving technique is very practical in many environments including server-farms and web-hosting facilities where space constraints are crucial.
Additional hardware solutions include intermediate routers and cables that increase the distance that may separate a user and a remote computer. These solutions can also increase the number of computers a user may control with one keyboard, monitor, and mouse. However this network is separate from existing LANs and Internet connections and may be hampered by a distance limitation.
The KVM switches have advantages over software solutions because they are not reliant upon the remote computer to function. If a system administrator needs to control and view a computer during “boot up” or to fix a problem with BIOS, the user can accomplish this via a remote keyboard, mouse and monitor linked via a KVM switch. Conversely, this would not be possible with a software solution.
Further, the KVM switch does not use processing power on the remote computer. From the point of view of both the controlled computer and the local computer, it is as if the video, mouse and keyboard are directly connected to the remote computer. Thus, no additional resources on the host computer are consumed.
Further, it is easier to make KVM switches that are operating system and machine independent. As long as the KVM ports are compatible with the keyboard, video and mouse connections, and with the output/input ports of the target computer, any KVM switch can be used, regardless of the operating system. With software solutions, a separate version of the software is generally needed if the user must control a variety of computers with a variety of operating systems.
Although KVM switches greatly improve the control of remote units, generally KVM switches rely on direct connections for sending signals from the host computer to the keyboard, video, and mouse that degrade over distances. For example, after a certain distance, the signal degradation affects the quality of the video signal transmitted. Therefore, if a system administrator or user needs access to a computer, the user still has to be within a certain distance of the computer.
In order to circumvent this transmission quality degradation over extended distances a KVM switch whereby the keyboard, video, and mouse signals are sent over standard Internet protocols or telephone connections maybe utilized. This allows any Internet or modem enabled device with a keyboard, video and mouse to control a remote computer regardless of the physical distance between a user computer and a remote device.
However, it has been proven in the art that the creation of such a system is much more difficult to implement than a direct wired KVM switch. In order to send video, keyboard, and monitor signals using a protocol such as those used on the Internet (e.g. TCP/IP, UDB) such analog signals must first be converted to digital signals. The digital signals, in uncompressed form, require a large bandwidth to be transmitted in near real-time. Generally, even high-speed connections such as cable and DSL are incapable of accommodating such bandwidth requirements. Furthermore, a majority of home users still connect to the Internet via a modem with further bandwidth limitations. Therefore, in order for such a device to be useful in these situations, the analog outputs of conventional monitors must be both converted to a digital signal and compressed.
Video compression takes advantage of the redundancies in video signals, both between successive frames of video, and within each individual frame. The transmission of a video signal from a computer monitor output generally has large amounts of both spatial and interframe redundancies. For example, in a near idle computer, the only change between successive frames of video might be the blinking of a cursor. Even as a user types a document, a majority of the screen does not change over periods of time.
Existing video compression standards are designed for common video applications. Generally, these compression systems are inappropriate for KVM switch application since these systems do not take into account specific KVM architecture. There exists a need in the art for a specialized KVM-specific algorithm capable of taking advantage of temporal redundancy, yet still capable of transmitting changes without a large loss of information.
Further, most forms of video compression known in the art require complicated calculations. For example, the MPEG standards use the discrete cosine transform as part of the compression algorithm. This standard relies on the recognition of “motion” between frames to calculate motion vectors to describe how portions of the image are affected over a period of time. These calculations are complicated and require a combination of expensive hardware, or result in extended transmission periods due to increased computation time.
Finally, many of the existing video compression techniques are lossy (i.e. they reduce the amount of information transmitted in order to reduce bandwidth). Typically, such lossy techniques either reduce the detail of an image or reduce the number of colors. Although reducing colors could be part of an adequate compression solution for computer monitor output, excessive reduction of images may yield a poor video transmission resulting in an illegible video reproduction. For example, if a computer user were attempting to use a word processor, reducing detail could lead to blurry or illegible text.
The field of compression and digitization of computer video through a video switch has seen explosive development over the years allowing the transfer of video data over extended distances at increased speed of transfer. For example, in a primitive form, in 1992 and 1993, Apple Computer developed a technology whereby one computer was controlled by another computer via emulation of keyboard and mouse protocols. This technology was implemented as part of a computer on a card “product.” The product consisted of a full computer developed on a single card that was designed to directly plug into a standard Macintosh computer. This Macintosh computer controlled the computer on a card via the keyboard and mouse emulation technologies. However, the video out from the computer on a card was routed to the Macintosh display in analog form. A digitization or compression method was not implemented, nor was a means for transmission of the video over great distances.
Other known methods in the art provide systems for converting VGA output to NTSC video. Such products (for example, TView Gold from Focus Enhancements of Campbell Calif.) allowed a computer's output to be viewed on a standard television. Over the years, numerous products have incorporated such technology whereby the output from a PC was digitized and displayed on a television screen. These products allowed the PC to be controlled via keyboard and mouse emulation. The user inputted keyboard and mouse signals into the apparatus, which manipulated and routed the signals to the PC. Although the system digitized video signals from a PC and completed some analysis to determine the size of the video, no compression methods were implemented.
Other products known in the art exist that convert video images from a Macintosh computer to a NTSC video output for display on a television screen. Generally, these products are cards that plug directly into a specific platform such as a Macintosh computer and are only capable of operating with this type of system. A common example of this product is called an L-TV. The L-TV product is designed such that it can read directly from the video memory of the Macintosh computer. In addition, some video compression techniques are used in the L-TV product such that only portions of the image that change between frames are retransmitted. However, by reading directly from video memory, L-TV only functions with a Macintosh computer. Other advances in the art are development of software based simulation systems.
Several patents are directed to the filed of compression and digitization of computer video signals. In addition, in certain instances, some of these systems operate in an environment of a user computer controlling a remote computer.
For example, Widergren U.S. Pat. No. 4,302,775 discloses a method for comparing sub blocks of an image between successive frames of video and only encoding the differences between the blocks for transmission. In Widergren, the block-by-block comparisons are completed in the transform domain. Thus the system requires extra computations necessary to compute the transform of the image thereby increasing the time necessary to complete the video compression. In order to obviate the problem and reduce transmission times, the disclosure of Widergren requires faster or more complex hardware. The present invention improves upon these time consuming extra computations by completing the block comparisons in the spatial domain. For example, the present invention utilizes a two-level thresholding method to ensure that the block comparisons are effective.
Santamäki et al. U.S. Pat. No. 4,717,957 teaches a method of caching previously occurring frames to decrease the necessary bandwidth for transmission of video. The process disclosed compares pixels from previous frames and only retransmits the changes between the pixels. Art disclosed before Santamäki compared a current frame of video with only the previous frame. Santamäki teaches a method that improves on previously existing art by adding a reference memory which may be used to store more than just the previous frame. Therefore, Santamäki teaches a method where the size of the cache is increased, thereby increasing the likelihood that a new frame of video will not have to be retransmitted.
The present invention improves on this disclosure by using two separate methods of storing previous frames and comparing the current frame of video with the previous frame. Furthermore, the present invention improves upon the efficiency of the cache comparisons by comparing the cyclic redundancy check for each block being compared.
Carr et al. U.S. Pat. No. 5,008,747 discloses a method for block-by-block comparison of sequential frames of video. Only changed pixels are retransmitted between frames of video. Carr et al. teaches a method whereby the changed pixels are stored in a matrix which is vector-quantized to one of a standard set of matrices. Thus Carr et al. discloses a video compression technique that uses temporal redundancies to reduce the data that must be transmitted. However, Carr et al. fails to disclose a method and apparatus capable of providing a reduced-time transmission of video. Further, Carr et al. fails to disclose a method of quantizing pixels before comparing frames. Thus the disclosures of Carr et al. would not be suited for remotely controlling a computer because it fails to teach methods that take into account noise that maybe introduced into the video through digitization errors.
Astle U.S. Pat. No. 5,552,832 discloses a camera that receives analog video signals and converts said signals to digital signals by implementing a microprocessor that divides the signals into blocks. The blocks of video are then classified and run-length encoded. Thus Astle discloses a video compression method that operates on blocks of pixels within an image. The present invention improves upon the compression techniques disclosed by taking advantage of temporal redundancies between images. Further, the present invention increases redundancy through noise elimination and a color lookup table.
Perholtz et al. U.S. Pat. No. 5,732,212 discloses a method for digitizing video signals for manipulation and transmission. The patent discloses a method whereby video raster signals from the data processing device are analyzed to determine the information displayed on a video display monitor attached to the data processing device. Perholtz et al. teaches a method for digitizing and compressing video. However, the method compresses video by analyzing the content of the video and sending said content. Thus in general, Perholtz does not teach a method in which the full graphical interface is displayed to the user. The present invention improves upon the disclosure of Perholtz by providing an improved graphical interface to the user. Further, the present invention improves upon this disclosure by compressing the video based upon spatial and temporal redundancies.
Frederick U.S. Pat. No. 5,757,424 discloses a system for high-resolution video conferencing without extreme demands on bandwidth. The system disclosed creates a mosaic image by sampling portions of a scene and combining those samples. This system allows for the transmission of video over low bandwidths. Frederick's system in general is used within a camera for transmitting video. Though Frederick teaches a way to reduce the data necessary for transmission, Frederick does not teach methods for comparing frames of video. In addition, Frederick does not teach a system whereby the video that must be sent is compressed using lossless compression. The present invention overcomes the limitations of Frederick's disclosures by using lossless compression in the spatial domain and two temporal redundancy checks. Further, the present invention teaches video compression in the context of controlling a remote computer rather than in the context of transmitting video from a camera.
Schneider U.S. Pat. No. 6,304,895 discloses a system for intelligently controlling a remotely located computer. Schneider further discloses a method of interframe block comparison where pixel values that even slightly change are retransmitted. This necessarily leads to retransmission of noisy pixels unnecessarily. In another embodiment, Schneider will retransmit an entire block if a threshold percentage of pixels within the block have changed.
For example, if all pixels in the current frame change from black to a dark gray due to noise introduced by the A/D conversion, all pixels will also be retransmitted unnecessarily because the total percentage (i.e. 100% of the pixels) would clearly exceed any predetermined percentage threshold. Schneider also fails to take into account legitimate changes. For example, an intended change to only a few pixels, e.g., 5 pixels, will be missed if the threshold is set to 6 pixels.
The present disclosure overcomes these shortcomings by recognizing minor changes due to noise, by implementing a more efficient calculation method and with a cache capable of storing previous blocks. Furthermore, the present disclosure recognizes significant changes (i.e. a pixel changing from black to white due to a cursor). In addition, slight color variations will be smoothed due to the color code and noise reduction methods of the present invention.
Pinkston U.S. Pat. No. 6,378,009 teaches a method of sending control, status and security functions over a network such as the Internet from one computer to another. Although Pinkston discloses a switching system that packetizes remote signals for the Internet, no video compression methods or conversions are disclosed. Instead, Pinkston teaches a method whereby a system administrator can access a KVM switch remotely over the Internet and control the switch. Therefore, in and of itself, Pinkston's disclosures would not allow a remote computer to be operated over a low-bandwidth connection.
The digitization of a video signal and its subsequent compression allows a computer to be controlled remotely using standard Internet protocols. The compression allows an interface to utilize digital encryption techniques known in the art. Non-digital KVM switches, in transmitting analog signals, do not allow or interface well with digital encryption schemes, such as 128-bit encryption. If a computer with sensitive information needs to be controlled from a remote location, there needs to be protection from potential hackers or competitors.
Therefore, what is needed is an Internet, LAN/WAN, or dial-up enabled KVM switch that allows for near real time transmission of compressed video. The compression must be efficient enough to transmit video in near real-time over modem bandwidths. However, the compression must not be too lossy, because the resulting image must be discernible. Finally, the KVM switch should work across multiple platforms (e.g. Macintosh, IBM compatible, and UNIX). Therefore, the switch cannot take advantage of platform dependent GDI calls, or similar system dependent codes which indicate when and where updates in the video are needed.
Based on the aforementioned disclosures and related technologies in the art, it is clear that there exists a need for a video compression method designed specifically for remotely monitoring and controlling a computer that is accurate and virtually provided in real-time. Furthermore, there exists a need in the art that allows for platform independent monitoring of computers, even at limited bandwidths provided by standard modem connections.